Partial WEKA Trace Format Support (without subagents) by SkiHatDuckie · Pull Request #972 · vllm-project/guidellm

SkiHatDuckie · 2026-07-28T19:23:03Z

Summary

Adds WEKA trace format support to GuideLLM. This PR includes basic handling and parsing of the format, updated tests and documentation, as well as some tweaking of preexisting formats. Subagent conversations, tool call events and nonlinear histories are still in active development, and will be added in a separate PR (ideally after #935 is merged).

This include new utilities for unwrapping JSON in utils/json_unwrap.py, used to parse and flatten conversations for trace replay.

Details

trace_weka.py (new)

Based off of this spec
Handles hash IDs and conversation-scoping
Currently missing:
- Subagent conversations
- Tool call events
- Nonlinear histories
- Handling of fields such as models, block_size, output_types, etc... (variable importance)
Accommodates for datasets which may not match the spec exactly by accepting any non-negative hash ID as valid and dropping partially filled hash IDs if they exist

test_trace_weka.py (new)

json_unwrap.py (new)

Added VirtualColumnLocation
Added functions construct_virtual_column_locations, unzip_virtual_column_locations, try_json_load, is_json_serializable and get_json_column_names

test_json_unwrap.py (new)

trace_common.py

Added conversation_id_column to TraceDataArgs
Rearranged helper function to fit new required checks
Added/updated helper functions to handle wrapped columns
Modified TraceExamplesIterable to handle localize relative timestamps to the conversation (if one exists)
Added optional method reset to TraceFormatBase
Moved dataset loading and validation up to the deserializer instead of the examples iterable.

trace_mooncake.py

Switch to storing token blocks as set[tuple[int, ...]] instead of list[list[int]] for minor performance boost
Switch to storing hash IDs in a dictionary instead of a lookup table

test_trace_common.py, test_trace_minimal.py & test_trace_mooncake.py

Minor tweaks to helper functions
New test for input token length matching or bordering the hash ID block size in test_trace_mooncake.py
New test for virtual columns in test_trace_common.py

trace_replay.md

Added documentation for weka
Included a warning about WIP features

datasets.md

Updated --data kind options list

Test Plan

tox -e lint-check && tox -e type-check
tox -e test-unit

Example datasets:
semianalysisai/cc-traces-weka-no-subagents-051226

Related Issues

Partial completion of #967

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes code generated or substantially modified by an AI agent
Includes tests generated or substantially modified by an AI agent

NOTE: the Generated-by or Assisted-by trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's DEVELOPING.md file.

git log

commit 8868aaa
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 13:57:55 2026 -0400

Add files

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 75371ff
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 14:05:52 2026 -0400

Update field description

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit e049ce8
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 14:18:54 2026 -0400

Add stubs

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit dc1920e
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 14:19:39 2026 -0400

Add missing trailing whitespace

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit a965b40
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jun 30 16:32:54 2026 -0400

Implement trace format & args

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 60cfcbd
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 11:39:42 2026 -0400

Add tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 2e1d969
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 12:12:17 2026 -0400

Test prompts matching or bordering block size

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit f28a347
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 13:20:29 2026 -0400

Update tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 1c37aaf
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 13:31:56 2026 -0400

Update trace format docs

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 7fef74a
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 1 14:10:31 2026 -0400

Update trace_replay.md

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 3450fc3
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Mon Jul 13 13:46:03 2026 -0400

Reapir tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 4edc5da
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 15 13:52:15 2026 -0400

Indicate private functions

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 58d8dab
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 15 13:56:37 2026 -0400

Update trace_replay.md

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 581f89b
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 15 13:58:55 2026 -0400

Make mdformat happy

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 4d7f58d
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 21 15:59:50 2026 -0400

Add json utils + impl in trace_common

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit ee391f0
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 21 16:20:35 2026 -0400

Refactor

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 16758aa
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 21 16:37:37 2026 -0400

Handle unserializable python dicts

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit fdb864f
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 22 12:00:52 2026 -0400

Improve json_unwrap tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 4aec222
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Wed Jul 22 12:45:52 2026 -0400

Improve trace_common test coverage

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 8310217
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Mon Jul 27 14:35:22 2026 -0400

Handle multi-conversation datasets

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit c1e2312
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Mon Jul 27 17:19:45 2026 -0400

Store token blocks as tuples

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 8622c0d
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 07:50:51 2026 -0400

Repair tests

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 9d04e2b
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 10:28:24 2026 -0400

Tolerate partially filled hash ID

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit a8872bb
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 11:40:27 2026 -0400

Improve test coverage

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit de082bb
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 12:02:54 2026 -0400

Fix linting

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit a833784
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 14:25:33 2026 -0400

Update docs

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 5fe0d30
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Tue Jul 28 14:54:45 2026 -0400

Fix typo

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 7ee289d
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Thu Jul 30 09:08:03 2026 -0400

Set conv_start_ts outside the loop

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 23129dc
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Thu Jul 30 09:14:50 2026 -0400

Move create_prompt_from_hash_ids to trace_common

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit f7d9c93
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Thu Jul 30 13:55:23 2026 -0400

Update docs

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 4afe96f
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Thu Jul 30 14:01:11 2026 -0400

Rename ColumnSearchResult field

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit d53490c
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Thu Jul 30 14:57:12 2026 -0400

Update guide

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit dfa94c6
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Thu Jul 30 15:18:20 2026 -0400

Try loading json columns if type `str`

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit a1002d1
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Thu Jul 30 15:23:14 2026 -0400

Handle final prompt if `prompt` is empty

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit f44b6fd
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Thu Jul 30 15:48:04 2026 -0400

Change final prompt again

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 0cabc16
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Thu Jul 30 16:09:39 2026 -0400

Change defaults to WEKA spec names

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit 387b5fb
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Fri Jul 31 12:12:58 2026 -0400

Patch dataset edge cases

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

commit b08662b
Author: SkiHatDuckie SkiHatDuckie@gmail.com
Date: Fri Jul 31 12:21:45 2026 -0400

Remove debug print

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Signed-off-by: SkiHatDuckie SkiHatDuckie@gmail.com

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

jaredoconnell

Looks good. I have a few comments.

Does this automatically support column names t / in / out? Sources like this state that it's what the spec is.

dbutenhof

Just a few initial comments. I haven't gotten far: and after spending most of the morning wrestling with AIPCC and GitLab workflows, I'm now heading out to hike (in the rain) before continuing this afternoon. Figured I might as well drop what I already got ...

SkiHatDuckie · 2026-07-29T16:15:28Z

Does this automatically support column names t / in / out? Sources like this state that it's what the spec is.

Currently no. I'll change the defaults for WEKA so that it fits the more common names.

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

dbutenhof

Just a quick diff pass for the moment ...

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

jaredoconnell

I left a comment on one of my prior comments.

Also, there is still a situation where a JSON string of a list causes a failure. Since it's hard to reproduce that, here is a test you can add to test_trace_common.py to validate your future fix:

   @pytest.mark.sanity
    def test_loads_requests_column_stored_as_json_string(
        self, tmp_path: Path, deserializer
    ):
        """Unwrap when the wrapper column is a JSON string of a list, not a native list.

        ## WRITTEN BY AI ##
        """
        trace = write_trace(
            tmp_path,
            '{"requests": "[{\\"timestamp\\": 0, \\"input_length\\": 10,'
            ' \\"output_length\\": 5}, {\\"timestamp\\": 1, \\"input_length\\": 20,'
            ' \\"output_length\\": 10}]"}\n',
        )
        ds = self.deserialize(deserializer, trace)
        rows = list(ds)
        assert len(rows) == 2
        assert rows[0]["prompt_tokens_count"] == 10
        assert rows[0]["output_tokens_count"] == 5
        assert rows[1]["prompt_tokens_count"] == 20
        assert rows[1]["output_tokens_count"] == 10

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

SkiHatDuckie added 27 commits July 28, 2026 14:27

Add files

8868aaa

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update field description

75371ff

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Add stubs

e049ce8

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Add missing trailing whitespace

dc1920e

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Implement trace format & args

a965b40

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Add tests

60cfcbd

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Test prompts matching or bordering block size

2e1d969

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update tests

f28a347

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update trace format docs

1c37aaf

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update trace_replay.md

7fef74a

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Reapir tests

3450fc3

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Indicate private functions

4edc5da

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update trace_replay.md

58d8dab

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Make mdformat happy

581f89b

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Add json utils + impl in trace_common

4d7f58d

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Refactor

ee391f0

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Handle unserializable python dicts

16758aa

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Improve json_unwrap tests

fdb864f

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Improve trace_common test coverage

4aec222

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Handle multi-conversation datasets

8310217

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Store token blocks as tuples

c1e2312

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Repair tests

8622c0d

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Tolerate partially filled hash ID

9d04e2b

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Improve test coverage

a8872bb

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Fix linting

de082bb

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update docs

a833784

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Fix typo

5fe0d30

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

dbutenhof assigned SkiHatDuckie Jul 29, 2026

dbutenhof added internal filed by core contributor or associate priority-critical labels Jul 29, 2026

dbutenhof added the feature Represents a new user-visible feature label Jul 29, 2026

dbutenhof added this to the v0.8.0 milestone Jul 29, 2026

dbutenhof requested review from dbutenhof, jaredoconnell and sjmonson July 29, 2026 14:51

sjmonson mentioned this pull request Jul 29, 2026

Subagent Conversation Support #935

Open

3 tasks

jaredoconnell requested changes Jul 29, 2026

View reviewed changes

Comment thread src/guidellm/data/deserializers/trace_weka.py Outdated

Comment thread src/guidellm/data/deserializers/trace_common.py Outdated

Comment thread src/guidellm/data/deserializers/trace_common.py Outdated

Comment thread src/guidellm/data/deserializers/trace_weka.py Outdated

dbutenhof reviewed Jul 29, 2026

View reviewed changes

Comment thread docs/guides/trace_replay.md Outdated

Comment thread docs/guides/trace_replay.md Outdated

Comment thread src/guidellm/data/deserializers/trace_common.py Outdated

SkiHatDuckie added 4 commits July 30, 2026 09:08

Set conv_start_ts outside the loop

7ee289d

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Move create_prompt_from_hash_ids to trace_common

23129dc

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Update docs

f7d9c93

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Rename ColumnSearchResult field

4afe96f

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

dbutenhof reviewed Jul 30, 2026

View reviewed changes

Comment thread docs/guides/trace_replay.md Outdated

SkiHatDuckie added 5 commits July 30, 2026 14:57

Update guide

d53490c

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Try loading json columns if type str

dfa94c6

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Handle final prompt if prompt is empty

a1002d1

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Change final prompt again

f44b6fd

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Change defaults to WEKA spec names

0cabc16

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

jaredoconnell requested changes Jul 30, 2026

View reviewed changes

SkiHatDuckie added 2 commits July 31, 2026 12:12

Patch dataset edge cases

387b5fb

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Remove debug print

b08662b

Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Partial WEKA Trace Format Support (without subagents) - #972

Partial WEKA Trace Format Support (without subagents)#972
SkiHatDuckie wants to merge 38 commits into
vllm-project:mainfrom
SkiHatDuckie:weka-trace

SkiHatDuckie commented Jul 28, 2026 •

edited by github-actions Bot

Loading

Uh oh!

jaredoconnell left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dbutenhof left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SkiHatDuckie commented Jul 29, 2026

Uh oh!

dbutenhof left a comment

Uh oh!

Uh oh!

jaredoconnell left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

SkiHatDuckie commented Jul 28, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Test Plan

Related Issues

Use of AI

git log

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SkiHatDuckie commented Jul 29, 2026

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SkiHatDuckie commented Jul 28, 2026 •

edited by github-actions Bot

Loading